The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification

Abstract In order to simplify sentences, several rewriting operations can be performed, such as replacing complex words per simpler synonyms, deleting unnecessary information, and splitting long sentences. Despite this multi-operation nature, evaluation of automatic simplification systems relies on metrics that moderately correlate with human judgments the simplicity achieved by executing specific (e.g., gain based lexical replacements). article, we investigate how well existing assess sentence-level simplifications where multiple may have been applied which, therefore, require more general judgments. For that, first collect a new reliable data set for evaluating correlation overall simplicity. Second, conduct meta-evaluation in Text Simplification, using our (and other data) analyze variation between metrics’ scores across three dimensions: perceived level, system type, references used computation. We show these aspects affect correlations and, particular, highlight limitations commonly operation-specific metrics. Finally, findings, propose recommendations simplifications, suggesting which compute interpret their scores.